Zero-Shot Robustification of Zero-Shot Models With Foundation Models

Zero-shot inference is a powerful paradigm that enables the use of large pretrained models for downstream classification tasks without further training. However, these models are vulnerable to inherited biases that can impact their performance. The traditional solution is fine-tuning, but this undermines the key advantage of pretrained models, which is their ability to be used out-of-the-box. We propose RoboShot, a method that improves the robustness of pretrained model embeddings in a fully zero-shot fashion. First, we use zero-shot language models (LMs) to obtain useful insights from task descriptions. These insights are embedded and used to remove harmful and boost useful components in embeddings -- without any supervision. Theoretically, we provide a simple and tractable model for biases in zero-shot embeddings and give a result characterizing under what conditions our approach can boost performance. Empirically, we evaluate RoboShot on nine image and NLP classification tasks and show an average improvement of 15.98% over several zero-shot baselines. Additionally, we demonstrate that RoboShot is compatible with a variety of pretrained and language models

IL4I1 Is a Novel Regulator of M2 Macrophage Polarization That Can Inhibit T Cell Activation via L-Tryptophan and Arginine Depletion and IL-10 Production

<div><p>Interleukin 4-induced gene-1 (IL4I1) was initially described as an early IL-4-inducible gene in B cells. IL4I1 protein can inhibit T cell proliferation by releasing its enzymatic catabolite, H₂O₂, and this effect is associated with transient down-regulation of T cell CD3 receptor-zeta (TCRζ) expression. Herein, we show that IL4I1 contributes to the regulation of macrophage programming. We found that expression of IL4I1 increased during bone marrow-derived macrophage (BMDM) differentiation, expression of IL4I1 is much higher in primary macrophages than monocytes, and IL4I1 expression in BMDMs could be induced by Th1 and Th2 cytokines in two different patterns. Gene expression analysis revealed that overexpression of IL4I1 drove the expression of M2 markers (Fizz1, Arg1, YM-1, MR) and inhibited the expression of M1-associated cytokines. Conversely, knockdown of IL4I1 by siRNA resulted in opposite effects, and also attenuated STAT-3 and STAT-6 phosphorylation. Furthermore, IL4I1 produced by macrophages catalyzed L-tryptophan degradation, while <i>levo</i>-1-methyl-tryptophan (L-1-MT), but not <i>dextro</i>-1-methyl-tryptophan, partially rescued IL4I1-dependent inhibition of T cell activation. Other inhibitors, such as diphenylene iodonium (DPI), an anti-IL-10Rα blocking antibody, and a nitric oxide synthase inhibitor, NG-monomethyl-L-arginine, also had this effect. Overall, our findings indicate that IL4I1 promotes an enhanced M2 functional phenotype, which is most likely associated with the phosphorylation of STAT-6 and STAT-3. Moreover, DPI, L-1-MT, NG-monomethyl-L-arginine, and anti-IL-10Rα blocking antibody were all found to be effective IL4I1 inhibitors <i>in vitro</i>.</p></div

Effective inhibitors of IL4I1 include L-1-MT, DPI, anti-IL-10Rα blocking antibody, and L-NMMA.

<p>BMDMs infected with IL4I1-recombinant or control retrovirus were pretreated with the inhibitors mentioned above or were untreated for 4 h. CFSE-labeled splenocytes from DO11.10 mice (n = 6) were cultured in a mixture of IL4I1-overexpressing macrophage-derived supernatant and conventional T cell medium (1:1) with 1 μg/mL soluble OVA_323–339 for 72 h. Cell division monitored based on levels of CFSE dilution that were measured using flow cytometry (A–B). Data are representative of three independent experiments. Levels of IFN-γ protein expression from the culture supernatants described above from DO11.10 splenocyte cultures were assayed by ELISA (C). Representative data are presented as means ± S.D. of three independent experiments. Significance was calculated by two-way ANOVA with multiple comparison post test (Bonferroni). Asterisks indicate significant differences compared to controls, or between two conditions that are linked by a bar; *p<0.05, **p<0.01, ***p<0.001.</p

Expression of IL4I1 throughout macrophage differentiation.

<p>BM cells at day 0 or during macrophage differentiation in the presence of M-CSF (20 ng/mL) for the indicated periods of time were collected. Levels of IL4I1 gene and IL4I1 protein expression were assessed by q-PCR (A) and western blotting (B), respectively. IL4I1 expression in medium (m) and total cellular proteins (c) from BMDMs were assessed by western blotting (C). Levels of IL4I1 gene and IL4I1 protein expression in primary monocytes and macrophages were assessed by q-PCR (D) and western blotting (E), respectively. All representative data are presented as means ± S.D. for three independent experiments. Significance was calculated by two tailed unpaired Student's <i>t</i>-test. Asterisks indicate significant differences compared to BM cells at day 0; *p<0.05, **p<0.01, ***p<0.001.</p

Induction of IL4I1 expression in macrophages.

<p>BMDMs were treated with LPS (100 ng/mL), IFN-γ (15 ng/mL), poly(I:C) (1 μg/mL), CpG (0.3 μM), or IL-4 (10 ng/mL) for the indicated amounts of time or were left untreated. Levels of IL4I1 expression were assayed by q-PCR (A–C, G–H) and western blotting (D–F, I–J), respectively. β-actin was used as a loading control throughout. Representative data are presented as means ± S.D. of three or four independent experiments. Significance was calculated by two tailed unpaired Student's <i>t</i>-test. Asterisks indicate significant differences compared to untreated cells; *p<0.05, **p<0.01, ***p<0.001.</p

IL4I1 limits acquisition of M1 phenotypes.

<p>BMDMs were infected with IL4I1-recombinant or control retrovirus, then were either untreated or treated with LPS (100 ng/mL) for 24 h. Levels of TNF-α, IL-1β, IL-12p40, and iNOS mRNA transcript expression were assayed by q-PCR (A) and those of TNF-α, IL-1β, and IL-12p40 protein expression were assayed by ELISA (C). BMDMs were transfected with siRNA against IL4I1 or scrambled siRNA for 24 h, then were untreated or treated with LPS (100 ng/mL) for 24 h. Levels of TNF-α, IL-1β, IL-12p40, and iNOS gene expression were assayed by q-PCR (B) and those of TNF-α, IL-1β, and IL-12p40 protein expression were assayed by ELISA (D). Representative data are presented as means ± S.D. of five independent experiments. Significance was calculated by one-way ANOVA with multiple comparison post test (Bonferroni). Asterisks indicate statistically significant differences, either compared to control, or between two conditions that are linked by a bar; *p<0.05, **p<0.01, ***p<0.001.</p